Increase Bot Up Time through Experimentation for Robotic Process Automation Platform

​

This project is intended to list out experiments to increase the login success % for data scraping bots.

​

Background about it: 

​

Robotic process automation platform at HighRadius has a 1000+ data aggregation bots repository setup across 100+ enterprise clients. 

​

What is a data aggregation bot ? : Bot that logs into a retail portal like Walmart or amazon on a daily basis and downloads business data like claims, PODs, Invoices and supplies it back to core products of Highradius products for data processing.

​

What is the problem ?

​

Monthly Login success % of these data aggregation bots currently stand at 80%. Overall business goal is to improve it from 80% to 90% to ensure continuous value delivery for what is being paid to Highradius by enterprise clients.

​

P.S. Login Success % = #Actual Logins/#Scheduled Logins

​

Post historical data analysis, it is identified that below are 2 key reasons for the 20% failures.

​

1. Retail portal Passwords Expiry contributes to 15% of failures
2. Unknown Login Failure issue contributing to remaining 5% of failures

​

Overview

​

Given above, we are planning to tackle the problem of portal password expiry by understanding through experiments why the client users fail to update the passwords before they expire even though Highradius robotic process automation platform is notifying them proactively.

​

Outcome : Increase in login success % by 10% across data aggregation bots

Hypothesis

If we proactively display in app popup for client users to let them know about portal passwords which are expired and need their attention with clear CTA to either

• Review Passwords : To reset portal passwords
• Notify Admin : Send in app communication to users who have access to update it.

​

Then our login success % will improve by 10% due to clear communication of the underlying problem right onto their face & CTA being the only focus.

Goal of the Experiment

Goal of the experiment is to validate the improvement in login success % metric by proactively displaying the in-app popup to reset passwords/notify admin as and when user logs into the product module tab.

​

Improvement in login success % going to drive the direct value to business given if data aggregation bots are able to login with a higher success rate, it will lead to a high level of data capture rate and high value for our enterprise clients.

​

Success Metrics

​

Best Case Scenario : Increase in login success % >10%

Worst Case Scenario : Increase in login success % >5% & <7%  

​

Experiment Design

What are we testing ?

​

Given, we have seen below in the past across H2 2023, that login success % across data aggregation bots stand at around ~80%

​

Hence we have planned to show proactive app popup to existing client users to nudge them to update passwords/notify admin for the same. 

​

Mockups : 

​

​

Variation Design

​

We will be introducing the new popup within claim tab with CTA

• Review Passwords 
• Notify Admin

​

Control Version copy will have Review Passwords as the primary CTA

​

Test Version copy will have Notify Admin as the primary CTA

​

Audience & Sample Size

​

This popup will be visible to analysts and managers currently having access to the claim module within the Highradius product.

​

Hence there will be 2 type of client users : Analyst & Managers

​

Sample Size : Above will be visible to cohort of 20 most impacted clients

​

Control : 20 (Primarily for managers)

Variation 1 : 20 (Primarily for analysts)

​

Duration of the Test : Above will be live for next 3 months since majority of passwords typically have time duration of 3 months

Implementation & A/A Test

​

Implementation Specs  : ARPA-Portal Credentials Expiry-250324-102226.pdf

A/A Test : Experiment will be run on the Prod Environment of Highradius. Both copies are live for 5 days. Here are the results

​

​

Given above, we can say that A/A test is successful across  the platform

Post Experiment

Experiment Results

​

Release Decision

​

Copy 1 makes sense to be shown for managers since 80% of managers who open the copy reset the passwords 

Copy 2 makes sense to be shown to analysts since 60% of analysts who open the copy actually notified the admins, it means that majority of analysts do not have access to update passwords themselves

​

To summarize this experiment can be scaled for rest of the enterprise clients as well given the results but CTA needs to be shown to the users based on their security roles setup in Highradius

Learnings

​

1. Password Expiry User notifications should mandatorily involve client manager persona in the To list.
2. Majorly users to whom password expiry should be communicated is not setup correctly today leading to drop in login success %
3. Review list of users accountable for password updates for each client is mandatory

Next Steps

​

1. Outreach plan via Customer support team to identify and segregate the manager vs analyst users setup within Highradius for each client - ETA 2 weeks from now
2. Include the right set of users in the notification list identified from step above - ETA 3 weeks from now
3. Measure the login success rate weekly post release of the feature to check if needle is moving the right direction - 4 weeks from now
4. Plan for a feature release based on experiment results with different CTA based on user role - 1 month from now

Stakeholder Management

​

​